On the “Degrees of Freedom” of the Lasso

We study the degrees of freedom of the Lasso in the framework of Stein’s unbiased risk estimation (SURE). We show that the number of non-zero coefficients is an unbiased estimate for the degrees of freedom of the Lasso—a conclusion that requires no special assumption on the predictors. Our analysis also provides mathematical support for a related conjecture by Efron et al. (2004). As an application, various model selection criteria—Cp, AIC and BIC—are defined, which, along with the LARS algorithm, provide a principled and efficient approach to obtaining the optimal Lasso fit with the computational efforts of a single ordinary least-squares fit. We propose the use of BIC-Lasso shrinkage if the Lasso is primarily used as a variable selection method. ∗Department of Statistics, Stanford University, Stanford, CA 94305. Email: [email protected]. †Department of Statistics and Department of Health Research & Policy, Stanford University, Stanford, CA 94305. Email: [email protected]. ‡Department of Health Research & Policy and Department of Statistics, Stanford University, Stanford, CA 94305. Email: [email protected].